Field
Embodiments of the present invention generally relate to the field of network security techniques. In particular, various embodiments relate to scanning network traffic logs retrospectively to detect threats missed during one or more prior signature-based scans.
Description of the Related Art
Network security appliances (e.g., firewalls, intrusion prevention systems (IPS), anti-virus (AV) devices and unified threat management (UTM) appliances) are deployed at the borders of networks to scan network traffic going through networks that are managed by the network security devices. Once a threat in the network traffic is detected by the network security devices, the network security devices may take an action to protect the networks (e.g., blocking the network traffic, recording the activity of the network traffic in a log, quarantining an associated resource, sending a message to the network administrator). The network security devices may also maintain network traffic logs to record part or all network activities observed within the networks.
The primary method of detecting network security threats is signature-based scanning. A detection engine of a network security device may scan network traffic in real time based on a signature database that can be accessed by the network security device locally or remotely. The network security device may also send the local network traffic log to a central or cloud-based log management system for additional scanning based on a more powerful signature database managed by the central or cloud-based log management system. For signature-based network security devices, a good signature database is critical for detecting threats timely and effectively. However, the signature creation process suffers from several delays, including a threat discovery delay that represents the time period between when a threat is encountered in the wild and when it is submitted to a security vendor, a threat identification delay that represents the time required to confirm that a potential threat is indeed an actual threat, a signature creation delay that represents the time required to create a signature that can properly identify the threat, a signature testing delay that represents the time required to verify that the signature does not trigger a false positive and a signature distribution delay that represents the time for the update package containing the new signature to be received by subscribing network security devices. Due to these delays, security threats are often missed by signature-based security solutions within the first hours, days or even weeks after the threats have first been encountered. To solve this problem, a host-based security solution (e.g., antivirus (AV) software) may periodically scan the entire computer with the latest signatures to catch threats missed by a network-based security solution. Unfortunately, network-based security appliances do not have this kind of luxury, since, among other things, it is impractical to store all the files or contents that have previously been scanned in order to scan them again later.